Motion direction is represented as a bimodal probability distribution in the human visual cortex.

Humans infer motion direction from noisy sensory signals. We hypothesize that to make these inferences more precise, the visual system computes motion direction not only from velocity but also spatial orientation signals - a 'streak' created by moving objects. We implement this hypothesis in a Bayesian model, which quantifies knowledge with probability distributions, and test its predictions using psychophysics and fMRI. Using a probabilistic pattern-based analysis, we decode probability distributions of motion direction from trial-by-trial activity in the human visual cortex. Corroborating the predictions, the decoded distributions have a bimodal shape, with peaks that predict the direction and magnitude of behavioral errors. Interestingly, we observe similar bimodality in the distribution of the observers' behavioral responses across trials. Together, these results suggest that observers use spatial orientation signals when estimating motion direction. More broadly, our findings indicate that the cortical representation of low-level visual features, such as motion direction, can reflect a combination of several qualitatively distinct signals.

Subjective confidence reflects representation of Bayesian probability in cortex.

What gives rise to the human sense of confidence? Here we tested the Bayesian hypothesis that confidence is based on a probability distribution represented in neural population activity. We implemented several computational models of confidence and tested their predictions using psychophysics and functional magnetic resonance imaging. Using a generative model-based decoding technique, we extracted probability distributions from neural population activity in human visual cortex. We found that subjective confidence tracks the shape of the decoded distribution. That is, when sensory evidence was more precise, as indicated by the decoded distribution, observers reported higher levels of confidence. We furthermore found that neural activity in the insula, anterior cingulate and prefrontal cortex was linked to both the shape of the decoded distribution and reported confidence, in ways consistent with the Bayesian model. Altogether, our findings support recent statistical theories of confidence and suggest that probabilistic information guides the computation of one's sense of confidence.

Dual strategies in human confidence judgments.

Although confidence is commonly believed to be an essential element in decision-making, it remains unclear what gives rise to one's sense of confidence. Recent Bayesian theories propose that confidence is computed, in part, from the degree of uncertainty in sensory evidence. Alternatively, observers can use physical properties of the stimulus as a heuristic to confidence. In the current study, we developed ideal observer models for either hypothesis and compared their predictions against human data obtained from psychophysical experiments. Participants reported the orientation of a stimulus, and their confidence in this estimate, under varying levels of internal and external noise. As predicted by the Bayesian model, we found a consistent link between confidence and behavioral variability for a given stimulus orientation. Confidence was higher when orientation estimates were more precise, for both internal and external sources of noise. However, we observed the inverse pattern when comparing between stimulus orientations: although observers gave more precise orientation estimates for cardinal orientations (a phenomenon known as the oblique effect), they were more confident about oblique orientations. We show that these results are well explained by a strategy to confidence that is based on the perceived amount of noise in the stimulus. Altogether, our results suggest that confidence is not always computed from the degree of uncertainty in one's perceptual evidence but can instead be based on visual cues that function as simple Heuristics to confidence.

Probabilistic Representation in Human Visual Cortex Reflects Uncertainty in Serial Decisions.

How does the brain represent the reliability of its sensory evidence? Here, we test whether sensory uncertainty is encoded in cortical population activity as the width of a probability distribution, a hypothesis that lies at the heart of Bayesian models of neural coding. We probe the neural representation of uncertainty by capitalizing on a well-known behavioral bias called serial dependence. Human observers of either sex reported the orientation of stimuli presented in sequence, while activity in visual cortex was measured with fMRI. We decoded probability distributions from population-level activity and found that serial dependence effects in behavior are consistent with a statistically advantageous sensory integration strategy, in which uncertain sensory information is given less weight. More fundamentally, our results suggest that probability distributions decoded from human visual cortex reflect the sensory uncertainty that observers rely on in their decisions, providing critical evidence for Bayesian theories of perception.SIGNIFICANCE STATEMENT Virtually any decision that people make is based on uncertain and incomplete information. Although uncertainty plays a major role in decision-making, we have but a nascent understanding of its neural basis. Here, we probe the neural code of uncertainty by capitalizing on a well-known perceptual illusion. We developed a computational model to explain the illusion, and tested it in behavioral and neuroimaging experiments. This revealed that the illusion is not a mistake of perception, but rather reflects a rational decision under uncertainty. No less important, we discovered that the uncertainty that people use in this decision is represented in brain activity as the width of a probability distribution, providing critical evidence for current Bayesian theories of decision-making.

Optimality is critical when it comes to testing computation-level hypotheses.

We disagree with Rahnev & Denison (R&D) that optimality should be abandoned altogether. Rather, we argue that adopting a normative approach enables researchers to test hypotheses about the brain's computational goals, avoids just-so explanations, and offers insights into function that are simply inaccessible to the alternatives proposed by R&D.

Perceptual learning increases orientation sampling efficiency.

Visual orientation discrimination is known to improve with extensive training, but the mechanisms underlying this behavioral benefit remain poorly understood. Here, we examine the possibility that more reliable task performance could arise in part because observers learn to sample information from a larger portion of the stimulus. We used a variant of the classification image method in combination with a global orientation discrimination task to test whether a change in information sampling underlies training-based benefits in behavioral performance. The results revealed that decreases in orientation thresholds with perceptual learning were accompanied by increases in stimulus sampling. In particular, while stimulus sampling was restricted to the parafoveal, inner portion of the stimulus before training, we observed an outward spread of sampling after training. These results demonstrate that the benefits of perceptual learning may arise, in part, from a strategic increase in the efficiency with which the observer samples information from a visual stimulus.

Sensory uncertainty decoded from visual cortex predicts behavior.

Bayesian theories of neural coding propose that sensory uncertainty is represented by a probability distribution encoded in neural population activity, but direct neural evidence supporting this hypothesis is currently lacking. Using fMRI in combination with a generative model-based analysis, we found that probability distributions reflecting sensory uncertainty could reliably be estimated from human visual cortex and, moreover, that observers appeared to use knowledge of this uncertainty in their perceptual decisions.

A review of the mechanisms by which attentional feedback shapes visual selectivity.

The glut of information available for the brain to process at any given moment necessitates an efficient attentional system that can 'pick and choose' what information receives prioritized processing. A growing body of work, spanning numerous methodologies and species, reveals that one powerful way in which attending to an item separates the wheat from the chaff is by altering a basic response property in the brain: neuronal selectivity. Selectivity is a cornerstone response property, largely dictating our ability to represent and interact with the environment. Although it is likely that selectivity is altered throughout many brain areas, here we focus on how directing attention to an item affects selectivity in the visual system, where this response property is generally more well characterized. First, we review the neural architecture supporting selectivity, and then discuss the various changes that could occur in selectivity for an attended item. In a survey of the literature, spanning neurophysiology, neuroimaging and psychophysics, we reveal that there is general convergence regarding the manner with which selectivity is shaped by attentional feedback. In a nutshell, the literature suggests that the type of changes in selectivity that manifest appears to depend on the type of attention being deployed: whereas directing spatial attention towards an item only alters spatial selectivity, directing feature-based attention can alter the selectivity of attended features.

Reinstatement of associative memories in early visual cortex is signaled by the hippocampus.

The cortical reinstatement hypothesis of memory retrieval posits that content-specific cortical activity at encoding is reinstated at retrieval. Evidence for cortical reinstatement was found in higher-order sensory regions, reflecting reactivation of complex object-based information. However, it remains unclear whether the same detailed sensory, feature-based information perceived during encoding is subsequently reinstated in early sensory cortex and what the role of the hippocampus is in this process. In this study, we used a combination of visual psychophysics, functional neuroimaging, multivoxel pattern analysis, and a well controlled cued recall paradigm to address this issue. We found that the visual information human participants were retrieving could be predicted by the activation patterns in early visual cortex. Importantly, this reinstatement resembled the neural pattern elicited when participants viewed the visual stimuli passively, indicating shared representations between stimulus-driven activity and memory. Furthermore, hippocampal activity covaried with the strength of stimulus-specific cortical reinstatement on a trial-by-trial level during cued recall. These findings provide evidence for reinstatement of unique associative memories in early visual cortex and suggest that the hippocampus modulates the mnemonic strength of this reinstatement.

Perceptual learning selectively refines orientation representations in early visual cortex.

Although practice has long been known to improve perceptual performance, the neural basis of this improvement in humans remains unclear. Using fMRI in conjunction with a novel signal detection-based analysis, we show that extensive practice selectively enhances the neural representation of trained orientations in the human visual cortex. Twelve observers practiced discriminating small changes in the orientation of a laterally presented grating over 20 or more daily 1 h training sessions. Training on average led to a twofold improvement in discrimination sensitivity, specific to the trained orientation and the trained location, with minimal improvement found for untrained orthogonal orientations or for orientations presented in the untrained hemifield. We measured the strength of orientation-selective responses in individual voxels in early visual areas (V1-V4) using signal detection measures, both before and after training. Although the overall amplitude of the BOLD response was no greater after training, practice nonetheless specifically enhanced the neural representation of the trained orientation at the trained location. This training-specific enhancement of orientation-selective responses was observed in the primary visual cortex (V1) as well as higher extrastriate visual areas V2-V4, and moreover, reliably predicted individual differences in the behavioral effects of perceptual learning. These results demonstrate that extensive training can lead to targeted functional reorganization of the human visual cortex, refining the cortical representation of behaviorally relevant information.

Less is more: expectation sharpens representations in the primary visual cortex.

Prior expectations about the visual world facilitate perception by allowing us to quickly deduce plausible interpretations from noisy and ambiguous data. The neural mechanisms of this facilitation remain largely unclear. Here, we used functional magnetic resonance imaging (fMRI) and multivariate pattern analysis (MVPA) techniques to measure both the amplitude and representational content of neural activity in the early visual cortex of human volunteers. We find that while perceptual expectation reduces the neural response amplitude in the primary visual cortex (V1), it improves the stimulus representation in this area, as revealed by MVPA. This informational improvement was independent of attentional modulations by task relevance. Finally, the informational improvement in V1 correlated with subjects' behavioral improvement when the expected stimulus feature was relevant. These data suggest that expectation facilitates perception by sharpening sensory representations.

Attention reverses the effect of prediction in silencing sensory signals.

Predictive coding models suggest that predicted sensory signals are attenuated (silencing of prediction error). These models, though influential, are challenged by the fact that prediction sometimes seems to enhance rather than reduce sensory signals, as in the case of attentional cueing experiments. One possible explanation is that in these experiments, prediction (i.e., stimulus probability) is confounded with attention (i.e., task relevance), which is known to boost rather than reduce sensory signal. However, recent theoretical work on predictive coding inspires an alternative hypothesis and suggests that attention and prediction operate synergistically to improve the precision of perceptual inference. This model posits that attention leads to heightened weighting of sensory evidence, thereby reversing the sensory silencing by prediction. Here, we factorially manipulated attention and prediction in a functional magnetic resonance imaging study and distinguished between these 2 hypotheses. Our results support a predictive coding model wherein attention reverses the sensory attenuation of predicted signals.

Dual roles for spike signaling in cortical neural populations.

A prominent feature of signaling in cortical neurons is that of randomness in the action potential. The output of a typical pyramidal cell can be well fit with a Poisson model, and variations in the Poisson rate repeatedly have been shown to be correlated with stimuli. However while the rate provides a very useful characterization of neural spike data, it may not be the most fundamental description of the signaling code. Recent data showing γ frequency range multi-cell action potential correlations, together with spike timing dependent plasticity, are spurring a re-examination of the classical model, since precise timing codes imply that the generation of spikes is essentially deterministic. Could the observed Poisson randomness and timing determinism reflect two separate modes of communication, or do they somehow derive from a single process? We investigate in a timing-based model whether the apparent incompatibility between these probabilistic and deterministic observations may be resolved by examining how spikes could be used in the underlying neural circuits. The crucial component of this model draws on dual roles for spike signaling. In learning receptive fields from ensembles of inputs, spikes need to behave probabilistically, whereas for fast signaling of individual stimuli, the spikes need to behave deterministically. Our simulations show that this combination is possible if deterministic signals using γ latency coding are probabilistically routed through different members of a cortical cell population at different times. This model exhibits standard features characteristic of Poisson models such as orientation tuning and exponential interval histograms. In addition, it makes testable predictions that follow from the γ latency coding.

Attention improves encoding of task-relevant features in the human visual cortex.

When spatial attention is directed toward a particular stimulus, increased activity is commonly observed in corresponding locations of the visual cortex. Does this attentional increase in activity indicate improved processing of all features contained within the attended stimulus, or might spatial attention selectively enhance the features relevant to the observer's task? We used fMRI decoding methods to measure the strength of orientation-selective activity patterns in the human visual cortex while subjects performed either an orientation or contrast discrimination task, involving one of two laterally presented gratings. Greater overall BOLD activation with spatial attention was observed in visual cortical areas V1-V4 for both tasks. However, multivariate pattern analysis revealed that orientation-selective responses were enhanced by attention only when orientation was the task-relevant feature and not when the contrast of the grating had to be attended. In a second experiment, observers discriminated the orientation or color of a specific lateral grating. Here, orientation-selective responses were enhanced in both tasks, but color-selective responses were enhanced only when color was task relevant. In both experiments, task-specific enhancement of feature-selective activity was not confined to the attended stimulus location but instead spread to other locations in the visual field, suggesting the concurrent involvement of a global feature-based attentional mechanism. These results suggest that attention can be remarkably selective in its ability to enhance particular task-relevant features and further reveal that increases in overall BOLD amplitude are not necessarily accompanied by improved processing of stimulus information.

Predictive feedback can account for biphasic responses in the lateral geniculate nucleus.

Biphasic neural response properties, where the optimal stimulus for driving a neural response changes from one stimulus pattern to the opposite stimulus pattern over short periods of time, have been described in several visual areas, including lateral geniculate nucleus (LGN), primary visual cortex (V1), and middle temporal area (MT). We describe a hierarchical model of predictive coding and simulations that capture these temporal variations in neuronal response properties. We focus on the LGN-V1 circuit and find that after training on natural images the model exhibits the brain's LGN-V1 connectivity structure, in which the structure of V1 receptive fields is linked to the spatial alignment and properties of center-surround cells in the LGN. In addition, the spatio-temporal response profile of LGN model neurons is biphasic in structure, resembling the biphasic response structure of neurons in cat LGN. Moreover, the model displays a specific pattern of influence of feedback, where LGN receptive fields that are aligned over a simple cell receptive field zone of the same polarity decrease their responses while neurons of opposite polarity increase their responses with feedback. This phase-reversed pattern of influence was recently observed in neurophysiology. These results corroborate the idea that predictive feedback is a general coding strategy in the brain.

The scalable mammalian brain: emergent distributions of glia and neurons.

In this paper, we demonstrate that two characteristic properties of mammalian brains emerge when scaling-up modular, cortical structures. Firstly, the glia-to-neuron ratio is not constant across brains of different sizes: large mammalian brains have more glia per neuron than smaller brains. Our analyses suggest that if one assumes that glia number is proportional to wiring, a particular quantitative relationship emerges between brain size and glia-to-neuron ratio that fits the empirical data. Secondly, many authors have reported that the number of neurons underlying one mm(2) of mammalian cortex is remarkably constant, across both areas and species. Here, we will show that such a constancy emerges when enlarging modular, cortical brain structures. Our analyses thus corroborate recent studies on the mammalian brain as a scalable architecture, providing a possible mechanism to explain some of the principles, constancies and rules that hold across brains of different size.

Interactions between higher and lower visual areas improve shape selectivity of higher level neurons-explaining crowding phenomena.

Recent theories of visual perception propose that feedforward cortical processing enables rapid and automatic object categorizations, yet incorporates a limited amount of detail. Subsequent feedback processing highlights high-resolution representations in early visual areas and provides spatial detail. To verify this hypothesis, we separate the contributions of feedforward and feedback signals to the selectivity of cortical neurons in a neural network simulation that is modeled after the hierarchical feedforward-feedback organization of cortical areas. We find that in such a network the responses of high-level neurons can initially distinguish between low-resolution aspects of objects but are 'blind' to differences in detail. After several feedback-feedforward cycles of processing, however, they can also distinguish between objects that differ in detail. Moreover, we find that our model captures recent paradoxical results of crowding phenomena, showing that spatial detail that is lost in visual crowding is nevertheless able to evoke specific adaptation effects. Our results thus provide an existence proof of the feasibility of novel theoretical models and provide a mechanism to explain various psychophysical and physiological results.

Boundary assignment in a recurrent network architecture.

We describe a model and simulations of boundary assignment by cortical neurons, a process that assigns edges to figural image regions, as opposed to the background regions on the other side of the edge. The model is composed of several areas, resembling the hierarchical feedforward-feedback organization of areas in the visual cortex. In each successive area along the hierarchy, the visual image is represented at a coarser resolution. Model neurons tend to assign edges to convex image regions. Because of high spatial resolution, information about convexity is not immediately available to all neurons in lower-level areas. In higher-level areas, however, spatial resolution is low, and convexity is coded more reliably. Feedback connections propagate this information to the high-resolution neurons of lower-level visual areas, making it available at all network levels and at all spatial resolutions. The proposed connection scheme assigns edges faster and more reliable to objects than one with only horizontal connections. The model accounts for both psychophysical and neurophysiological data on figural assignment.

Learning receptive fields using predictive feedback.

Previously, it was suggested that feedback connections from higher- to lower-level areas carry predictions of lower-level neural activities, whereas feedforward connections carry the residual error between the predictions and the actual lower-level activities [Rao, R.P.N., Ballard, D.H., 1999. Nature Neuroscience 2, 79-87.]. A computational model implementing the hypothesis learned simple cell receptive fields when exposed to natural images. Here, we use predictive feedback to explain tuning properties in medial superior temporal area (MST). We implement the hypothesis using a new, biologically plausible, algorithm based on matching pursuit, which retains all the features of the previous implementation, including its ability to efficiently encode input. When presented with natural images, the model developed receptive field properties as found in primary visual cortex. In addition, when exposed to visual motion input resulting from movements through space, the model learned receptive field properties resembling those in MST. These results corroborate the idea that predictive feedback is a general principle used by the visual system to efficiently encode natural input.